Predicting a concept prediction for media

ABSTRACT

For predicting a concept prediction, a processor generates a review job that includes a review question and a corresponding expert concept prediction for an expert media set drawn from a media corpus. The processor generates media reviews from a plurality of reviewers using the review job for the expert media set. The processor extracts media features for the media of the media reviews. The processor trains a review model with the media reviews and corresponding media features. The processor predicts the concept prediction for media of the media corpus using the review model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/752,905 entitled “EXTRACTING SUBJECTIVE OPINIONS OF VIDEOS” andfiled on Oct. 30, 2018 for Jonathan Mora, the entire contents of whichis incorporated herein by reference.

FIELD

The subject matter disclosed herein relates to predicting subjectiveconcept predictions for media.

BACKGROUND

Large quantities of media are difficult to accurately review.

BRIEF DESCRIPTION OF THE DRAWINGS

A more particular description of the embodiments briefly described abovewill be rendered by reference to specific embodiments that areillustrated in the appended drawings. Understanding that these drawingsdepict only some embodiments and are not therefore to be considered tobe limiting of scope, the embodiments will be described and explainedwith additional specificity and detail through the use of theaccompanying drawings, in which:

FIG. 1A is a schematic block diagram illustrating one embodiment of areview process;

FIG. 1B is a schematic block diagram illustrating one alternateembodiment of a review process;

FIG. 1C is a schematic block diagram illustrating one embodiment of amedia corpus;

FIG. 2 is a schematic block diagram illustrating one embodiment of areview system;

FIG. 3A is a schematic block diagram illustrating one embodiment of areview job database;

FIG. 3B is a schematic block diagram illustrating one embodiment of areview job;

FIG. 3C is a schematic block diagram illustrating one embodiment of areview database;

FIG. 3D is a schematic block diagram illustrating one embodiment of areview;

FIG. 3E is a schematic block diagram illustrating one embodiment of amedia review database;

FIG. 3F is a schematic block diagram illustrating one embodiment of amedia review;

FIG. 3G is a schematic block diagram illustrating one embodiment ofsystem data;

FIG. 3H is a schematic block diagram illustrating one embodiment ofconstrained optimization data;

FIG. 3I is a schematic block diagram illustrating one embodiment of apreference format;

FIG. 3J is a schematic block diagram illustrating one embodiment ofmedia features;

FIG. 4A is a schematic block diagram illustrating one embodiment of acomputer;

FIG. 4B is a schematic block diagram illustrating one embodiment of aneural network;

FIG. 5A is a flow chart diagram illustrating one embodiment of a modelgeneration method 520.

FIG. 5B is a flow chart diagram illustrating one embodiment of a mediaselection method;

FIG. 5C is a flow chart diagram illustrating one embodiment of a reviewjob generation method;

FIG. 5D is a flow chart diagram illustrating one embodiment of a mediareview generation method;

FIG. 5E is a flow chart diagram illustrating one embodiment of a modeltraining method;

FIG. 5F is a flow chart diagram illustrating one embodiment of aconstrained optimization method;

FIG. 5G is a flow chart diagram illustrating one embodiment of a mediaselection method;

FIG. 5H is a flow chart diagram illustrating one embodiment of a mediafeature identification method; and

FIG. 5I is a flow chart diagram illustrating one embodiment of a mediabidding method.

DETAILED DESCRIPTION

As will be appreciated by one skilled in the art, aspects of theembodiments may be embodied as a system, method or program product.Accordingly, embodiments may take the form of an entirely hardwareembodiment, an entirely software embodiment (including firmware,resident software, micro-code, etc.) or an embodiment combining softwareand hardware aspects that may all generally be referred to herein as a“circuit,” “module” or “system.” Furthermore, embodiments may take theform of a program product embodied in one or more computer readablestorage devices storing machine readable code, computer readable code,and/or program code, referred hereafter as code. The storage devices maybe tangible, non-transitory, and/or non-transmission. The storagedevices may not embody signals. In a certain embodiment, the storagedevices only employ signals for accessing code.

Many of the functional units described in this specification have beenlabeled as modules, in order to more particularly emphasize theirimplementation independence. For example, a module may be implemented asa hardware circuit comprising custom Very Large Scale Integration (VLSI)circuits or gate arrays, off-the-shelf semiconductors such as logicchips, transistors, or other discrete components. A module may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices or thelike.

Modules may also be implemented in code and/or software for execution byvarious types of processors. An identified module of code may, forinstance, comprise one or more physical or logical blocks of executablecode which may, for instance, be organized as an object, procedure, orfunction. Nevertheless, the executables of an identified module need notbe physically located together, but may comprise disparate instructionsstored in different locations which, when joined logically together,comprise the module and achieve the stated purpose for the module.

Indeed, a module of code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be identified and illustrated hereinwithin modules, and may be embodied in any suitable form and organizedwithin any suitable type of data structure. The operational data may becollected as a single data set, or may be distributed over differentlocations including over different computer readable storage devices.Where a module or portions of a module are implemented in software, thesoftware portions are stored on one or more computer readable storagedevices.

Any combination of one or more computer readable medium may be utilized.The computer readable medium may be a computer readable storage medium.The computer readable storage medium may be a storage device storing thecode. The storage device may be, for example, but not limited to, anelectronic, magnetic, optical, electromagnetic, infrared, holographic,micromechanical, or semiconductor system, apparatus, or device, or anysuitable combination of the foregoing.

More specific examples (a non-exhaustive list) of the storage devicewould include the following: an electrical connection having one or morewires, a portable computer diskette, a hard disk, a random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Code for carrying out operations for embodiments may be written in anycombination of one or more programming languages including an objectoriented programming language such as Python, Ruby, R, Java, Scala, JavaScript, Smalltalk, C++, C #, Lisp, Clojure, PHP, or the like, andconventional procedural programming languages, such as the “C”programming language, or the like, and/or machine languages such asassembly languages. The code may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment. Thus, appearances of the phrases“in one embodiment,” “in an embodiment,” and similar language throughoutthis specification may, but do not necessarily, all refer to the sameembodiment, but mean “one or more but not all embodiments” unlessexpressly specified otherwise. The terms “including,” “comprising,”“having,” and variations thereof mean “including but not limited to,”unless expressly specified otherwise. An enumerated listing of itemsdoes not imply that any or all of the items are mutually exclusive,unless expressly specified otherwise. The terms “a,” “an,” and “the”also refer to “one or more” unless expressly specified otherwise. Theterm “and/or” indicates embodiments of one or more of the listedelements, with “A and/or B” indicating embodiments of element A alone,element B alone, or elements A and B taken together.

Furthermore, the described features, structures, or characteristics ofthe embodiments may be combined in any suitable manner. In the followingdescription, numerous specific details are provided, such as examples ofprogramming, software modules, user selections, network transactions,database queries, database structures, hardware modules, hardwarecircuits, hardware chips, etc., to provide a thorough understanding ofembodiments. One skilled in the relevant art will recognize, however,that embodiments may be practiced without one or more of the specificdetails, or with other methods, components, materials, and so forth. Inother instances, well-known structures, materials, or operations are notshown or described in detail to avoid obscuring aspects of anembodiment.

Aspects of the embodiments are described below with reference toschematic flowchart diagrams and/or schematic block diagrams of methods,apparatuses, systems, and program products according to embodiments. Itwill be understood that each block of the schematic flowchart diagramsand/or schematic block diagrams, and combinations of blocks in theschematic flowchart diagrams and/or schematic block diagrams, can beimplemented by code. This code may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the functions/acts specified in the schematic flowchartdiagrams and/or schematic block diagrams block or blocks.

The code may also be stored in a storage device that can direct acomputer, other programmable data processing apparatus, or other devicesto function in a particular manner, such that the instructions stored inthe storage device produce an article of manufacture includinginstructions which implement the function/act specified in the schematicflowchart diagrams and/or schematic block diagrams block or blocks.

The code may also be loaded onto a computer, other programmable dataprocessing apparatus, or other devices to cause a series of operationalsteps to be performed on the computer, other programmable apparatus orother devices to produce a computer implemented process such that thecode which execute on the computer or other programmable apparatusprovide processes for implementing the functions/acts specified in theflowchart and/or block diagram block or blocks.

The schematic flowchart diagrams and/or schematic block diagrams in theFigures illustrate the architecture, functionality, and operation ofpossible implementations of apparatuses, systems, methods and programproducts according to various embodiments. In this regard, each block inthe schematic flowchart diagrams and/or schematic block diagrams mayrepresent a module, segment, or portion of code, which comprises one ormore executable instructions of the code for implementing the specifiedlogical function(s).

It should also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in theFigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. Other steps and methods may be conceived that are equivalentin function, logic, or effect to one or more blocks, or portionsthereof, of the illustrated Figures.

Although various arrow types and line types may be employed in theflowchart and/or block diagrams, they are understood not to limit thescope of the corresponding embodiments. Indeed, some arrows or otherconnectors may be used to indicate only the logical flow of the depictedembodiment. For instance, an arrow may indicate a waiting or monitoringperiod of unspecified duration between enumerated steps of the depictedembodiment. It will also be noted that each block of the block diagramsand/or flowchart diagrams, and combinations of blocks in the blockdiagrams and/or flowchart diagrams, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts, or combinations of special purpose hardware and code.

The description of elements in each figure may refer to elements ofproceeding figures. Like numbers refer to like elements in all figures,including alternate embodiments of like elements.

Large numbers of media are difficult to accurately review. Theembodiments allow expert opinions to be scaled and automated usingmachine learning.

FIG. 1A is a schematic block diagram illustrating one embodiment of areview process 100 a. Many advertisers would like to present theiroffering to a wide audience, including in-front of streamed mediacontent, for example—video media on platforms such as YOUTUBE® orFACEBOOK®. The advertiser would like to present their offering in acontext that reflects their brand positioning. That context is bothsubjective and changes over time. In order to capture a subjective andchanging context for a brand, there is a need to translate the brand'spreferences into a media set 107 that will adhere to the brand's intent.Advertisements, promotions, offers, and the like may be presented alongwith presentations of a medium from the media set 107 to further theadvertiser's goals.

In the past, the media set 107 has been manually selected from the mediacorpus 101, and/or selected based on easily quantifiable criteria suchas the number of views, whitelists, blacklists, keyword targeting,and/or a number of likes. Unfortunately, such selection processesignored highly suitable media and/or focused on media that was in moredemand and so more costly. As a result, promotional opportunities withmedia that reflects a brand's context are regularly lost whilepromotional costs are often higher than needed.

The embodiments scale the review process 100 a and translate the manualreview process into an automated review process 100 a as will bedescribed hereafter. By automating the review process 100 a, theembodiments allow for a larger media corpus 101 to be effectivelyreviewed. As a result, more suitable media may be identified. Inaddition, more suitable and/or cost-effective media may be identified.Thus the embodiments improve the efficiency of computer systems inselecting the media set 107 from the media corpus 101 for advertisers.

The review process 100 a employs a review model 105 to review a mediacorpus 101 and select the media set 107. The review model 105 is trainedusing human review test results 109 so that the review model 105accurately reflects the opinions of human reviewers. The human reviewtest results 109 scale the opinions of experts such as advertisers to alarge number of reviewers. The reviewers then generate a sufficienttraining set of media reviews that can be used to train the review model105. As a result, the opinions of the few experts are scaled to theautomated review model 105.

The review model 105 automatically reviews media from the media corpus101 and generates a concept prediction for each medium. The conceptprediction predicts how closely the theme of a medium reflects a brandpositioning. For example, a medium about applying makeup may closelyreflect the brand positioning of a cosmetic product. The conceptprediction is used to select the media of the media set 107. Thus,promotions can be presented with the media of the media set 107 topromote and enhance a brand.

In one embodiment, the media of the media corpus 101 is initiallyfiltered to identify media that are more likely to reflect the brandpositioning. The review model 105 may then review this subset of themedia corpus 101 and predict the concept prediction for the subset ofthe media corpus 101.

FIG. 1B is a schematic block diagram illustrating one alternateembodiment of a review process 100 b. In the depicted embodiment, mediathat is about to be consumed is placed in a bid queue 121. For example,each video media that is about to be viewed on YOUTUBE® may be placed inthe bid queue 121. A medium 115 may be selected from the bid queue 121and reviewed by the review model 105. The review model 105 may predictthe concept prediction for the medium 115 and store the conceptprediction. The concept prediction may be used to generate a medium bid123 that accurately reflects the value of presenting a promotion withthe medium 115 to the advertiser.

FIG. 1C is a schematic block diagram illustrating one embodiment of themedia corpus 101. In the depicted embodiment, the media corpus 101includes a plurality of media 115. For simplicity, two media 115 areshown. However, the media corpus 101 may include many billions ofindividual medium 115. As a result, the media corpus 101 is too largefor any person or group to manually review. As a result, the generationof lists such as blacklists and whitelists does not scale for a largemedia corpus 101. In the past, easily quantifiable metrics such askeywords, views, and likes have been used to make selections of themedia 115. However, inaccuracies in the keywords and the like have oftenresulted in unsuitable media 115 being selected. In addition, media 115suitable for promoting a brand may not be tagged with keywords thatresult in the media's selection.

The embodiments may employ an expert media set 113 to generate a reviewjob that is used to generate a large number of reviewers trained in theopinions of the experts. The expert media set 113 may be a subset of themedia corpus 101. The reviewers then generate a large number of conceptpredictions from a reviewer media set 114 selected from the media corpus101 that is used to train the review model 105 as will be describedhereafter. The reviewer media set 114 may be a subset of the mediacorpus 101. The reviewer media set 114 may be 10 to 100 times or morelarger than the expert media set 113.

The expert media set 113 may be a subset of the media corpus 101. Theexpert media set 113 may be selected programmatically. For example, theexpert media set 113 may be selected using active learning. The expertmedia set 113 may be manually selected. For example, the expert mediaset 113 may be a curated set of media 115 that is regularly presented toadvertisers. Alternatively, the expert media set 113 may be randomlyselected from the media corpus 101. In a certain embodiment, the expertmedia set 113 is selected using the media filter 103.

FIG. 2 is a schematic block diagram illustrating one embodiment of areview system 150. The review system 150 may coordinate the expertise ofone or more experts 205, moderators 207, and reviewers 209 in generatingthe human review test results 109 that are used to train the reviewmodel 105. In the depicted embodiment, the expert 205, moderator 207,and reviewers 209 communicate with the reviewer server 201. In addition,the reviewers 209 may communicate with the production server 211.

One or more experts 205 share a collective subjective opinion of media115 in the media corpus 101. In one embodiment, the advertiser is theexpert 205. In addition, the experts 205 may be selected by theadvertiser. As a result, the experts 205 have a shared set of subjectivepreferences that reflect the advertiser's brand positioning.

The moderator 207 may be knowledgeable of the media corpus 101. In oneembodiment, the moderator 207 is an automated process. The reviewers 209may review media 115 from the media corpus 101 and answer objectivequestions about the media 115 to generate media reviews with conceptpredictions.

Unfortunately, the number of media 115 that should be reviewed togenerate the human review test results 109 sufficient to train thereview model 105 is typically larger than the number of media 115 theexperts 205 are collectively willing to view. For example, an advertiserexpert 205 may be willing to review the media 115 in the expert mediaset 113, but the expert media set 113 may not be large enough toeffectively train the review model 105. As a result, the embodiments useexpert concept predictions from the experts 205 using the expert mediaset 113 to select and train the plurality of reviewers 209. Theplurality of reviewers 209 may then review the reviewer media set 114and generate the human review test results 109 for training the reviewmodel 105 on the reviewer server 201. The review model 105 may then behosted on the production server 211 and review the media corpus 101and/or bid queue 121. In one embodiment, reviews from the reviewers 209may periodically be used to retrain and/or refine the review model 105.

FIG. 3A is a schematic block diagram illustrating one embodiment of areview job database 300. The review job database 300 may be organized asa data structure in a memory. In the depicted embodiment, the review jobdatabase 300 comprises a plurality of review jobs 301. In oneembodiment, the review job database 300 includes a promotion 319. Thepromotion 319 may record links to one or more promotional medium thatmay be presented along with a medium 115 to promote a brand's intent.The promotional media may include advertisements, coupons, universalresource locators (URL), and/or the like.

FIG. 3B is a schematic block diagram illustrating one embodiment of areview job 301. Each review job 301 may be based on the opinions of anexpert 205, including an expert concept prediction 313. In the depictedembodiment, the review job 301 includes a media identifier 303, anobjective description 305, a review question 307, a test question 311,the expert concept prediction 313, an answer description 315, and mediafeatures 317.

The media identifier 303 may identify a medium 115 such as a video,audio, text, an image, and the like. The objective description 305 maydescribe the medium 115. The review question 307 may elicit the expertconcept prediction 313 from the expert 205. For example, the reviewquestion 307 may ask “Is the current video representative of our brandidentity?”

In one embodiment, the test question 311 comprises a question thatvalidates the review question 307 and corresponding expert conceptprediction 313. The answer description 315 may record additional detailsfrom the expert 205 regarding the expert concept prediction 313. Forexample, the answer description 315 may explain why the current video isrepresentative of the brand identity. The media features 317 aredescribed hereafter in FIG. 3J.

The review job 301 organizes data for efficiently generating the expertconcept prediction 313 from an expert 205. Thus, the review job 301enables a computer to accurately capture the expert concept prediction313.

FIG. 3C is a schematic block diagram illustrating one embodiment of areview database 350. The review database 350 may be organized as a datastructure in a memory. In the depicted embodiment, the review database350 includes a plurality of reviews 351. The review database 350 mayalso include the promotion 319.

FIG. 3D is a schematic block diagram illustrating one embodiment of areview 351. The review 351 may be generated by a reviewer 209 inresponse to a medium 115 using the review job 301. In the depictedembodiment, the review 351 includes the media identifier 303 and one ormore concept predictions 309. The media identifier 303 identifies themedium 115. The concept prediction 309 records the reviewer's answer tothe review question 307 of the review job 301 regarding the medium 115.

FIG. 3E is a schematic block diagram illustrating one embodiment of amedia review database 370. The media review database 370 may beorganized as a data structure in a memory. In the depicted embodiment,the media review database 370 includes a plurality of media reviews 371.

FIG. 3F is a schematic block diagram illustrating one embodiment of amedia review 371. The media review 371 may aggregate a plurality ofreviews 351 for a medium 115. In the depicted embodiment, the mediareview 371 includes the media identifier 303, a consensus conceptprediction 373, and the media features 317. The media identifier 303identifies the medium 115. The consensus concept prediction 373 may bean average, a mean, a medium, a range, and the like of the conceptpredictions 309 of the plurality of reviews 351 for the medium 115.

FIG. 3G is a schematic block diagram illustrating one embodiment ofsystem data 390. The system data 390 may be organized as a datastructure in a memory. In the depicted embodiment, the system data 390includes model parameters 391, an accuracy threshold 393, a sentimentthreshold 394, and an agreement threshold 395. The model parameters 391may specify parameters for training the review model 105. The modelparameters 391 may include a model type, a model breadth, a model depth,a learning rate, a learning algorithm, a network geometry, and/or arecursion geometry.

The accuracy threshold 393 may be used to validate a review job 301,determine whether to regenerate the review job 301, and/or selectreviewers 209 as will be described hereafter. The accuracy threshold 393may be satisfied if the concept predictions 309 agrees with the expertconcept predictions 313. The accuracy threshold 393 may be a percentagein the range of 70 to 100 percent.

The sentiment threshold 394 may be used to determine if a reviewquestion 307 elicits a valid expert concept prediction 313. Thesentiment threshold 394 may be in the range of 60 to 100 percentpositive sentiment. The agreement threshold 395 may be used to determineif a review job 301 used by reviewers 209 sufficiently reflects theexpert concept prediction 313 of the expert 205. The agreement threshold395 may be a probability of a null hypothesis of less than five percent.In addition, the agreement threshold 395 may be a probability given aprior of in the range of 0.7 to 1.0.

FIG. 3H is a schematic block diagram illustrating one embodiment ofconstrained optimization data 220. The constrained optimization data 220may be organized as a data structure in a memory. The constrainedoptimization data 220 may be used to perform a constrained optimizationas will be described hereafter in FIG. 5B. In the depicted embodiment,the constrained optimization data 220 includes budget requirements 221,a placement number 223, a desired spend 225, an objective function 227,and a constrained optimization 229.

The budget requirements 221 may specify a promotional budget of thepromotion 319 for one or more specified time intervals. For example, thebudget requirements 221 may specify a maximum promotional budget for aday and/or for a month. The placement number 223 may specify a number ofmedia 115 with which to place the promotion 319. The desired spend 225may specify an aggregate cost of placing the promotion 319 with media115. The objective function 227 may relate the budget requirements 221,the placement number 223, the desired spend 225, and other factors. Theconstrained optimization 229 may specify values for one or moreparameters as will be described hereafter.

FIG. 3I is a schematic block diagram illustrating one embodiment of apreference format 240. The preference format 240 may be organized as adata structure in a memory. The preference format 240 may be generatedfrom an expert 205. In addition, the preference format 240 may be usedto generate a review job 301. In the depicted embodiment, the preferenceformat 240 includes the user identifier 241, a user profile 243, and oneor more review sets 261 that include the objective description 305, thereview question 307, and the expert concept prediction 313. In oneembodiment, the review sets 261 include the answer description 315.

FIG. 3J is a schematic block diagram illustrating one embodiment of themedia features 317. The media features 317 may be organized a datastructure in a memory. In the depicted embodiment, the media features317 include media metrics 353, a media category 247, a text description249, an image 251, text 253, a video 255, and audio 257. The mediametrics 353 may record a number of views, a number of likes, commentsentiments, keywords, and the like for the medium 115. The mediacategory 247 may specify one or more categories for the medium 115. Themedium 115 may be embodied in the image 251, text 253, video 255, and/oraudio 257.

The text description 249 may describe the medium 115. In one embodiment,the text description 249 includes keywords for the medium 115. The image251 may record an image medium 115. In addition, the image 251 mayrecord a representative image for the text 253, video 255 and/or audio257. The text 253 may record a text medium 115. In addition, the text253 may describe the image 251, the video 255, and/or the audio 257. Thevideo 255 may record a video medium 115. In addition, the video 255 maydescribe the image 251, the text 253, and/or the audio 257. The audio257 may record an audio medium 115. In addition, the audio 257 maydescribe the image 251, the text and 53, and/or the video 255.

FIG. 4A is a schematic block diagram illustrating one embodiment of acomputer 400. In the depicted embodiment, the computer 400 includes aprocessor 405, a memory 410, and communication hardware 415. The memory410 may include a semiconductor storage device, a hard disk drive, anoptical storage device, a micromechanical storage device, orcombinations thereof. The memory 410 may store code. The processor 405may execute the code. The communication hardware 415 may communicatewith other devices.

The processor 405 may train and/or execute the review model 105. Thereview model may be a Gradient Boosting Machine (GBM) model, a randomforest model, and the like. The processor 405 may employ supervisedlearning or unsupervised learning to train the review model 105.

FIG. 4B is a schematic block diagram illustrating one embodiment of aneural network 475. In the depicted embodiment, the neural network 475includes input neurons 450, hidden neurons 455, and output neurons 460.The neural network 475 may be organized as a convolutional neuralnetwork, a recurrent neural network, a long-short-term memory network,and the like.

The neural network 475 may be trained with training data. The trainingdata may include the media reviews 371 of their media review database370. The neural network 475 may be trained using one or more learningfunctions while applying the training data such as media features 317 tothe input neurons 450 for unsupervised learning. In addition, knownresult values such as corresponding concept predictions 309 may beidentified for the output neurons 460 for supervised learning.Subsequently, the neural network 465 may receive actual data such as themedia features 317 at the input neurons 450 and make a conceptprediction 309 at the output neurons 460 based on the actual data.

FIG. 5A is a flow chart diagram illustrating one embodiment of a modelgeneration method 500. The method 500 may generate the review model 105.The method 500 may be performed by the processor 405 and/or neuralnetwork 475.

The method 500 starts, and in one embodiment, the processor 405generates 501 the review job 301 comprising at least one review question307 and the corresponding expert concept prediction 313 for an expertmedia set 113 drawn from a media corpus 101. The review job 301 may begenerated 501 from one or more experts 205. In addition, the review job301 may be validated with one or more reviewers 209. The generation 501of the review job 301 is described in more detail in FIG. 5C.

The processor 405 may select 503 a media sourcing strategy. The mediasourcing strategy may be selected from the group consisting of a randomstrategy, an active learning strategy, and a targeted strategy. Therandom strategy selects media 115 at random. For example, the processor405 may randomly select one or more media identifiers 303 from the mediacorpus 101.

The active learning strategy selects media 115 for review based on theconcept prediction 309 from the reviewers 209 and/or interactive queriesdirected to reviewers 209. For example, the review model 105 maygenerate a concept prediction 309 for a candidate medium 115. Thecandidate medium 115 may be selected based on the concept prediction309. In addition, the processor 405 may present the review question 307for a candidate medium 115 to one or more reviewers 209 and receive theconcept prediction 309 from the reviewers 209. The processor 405 mayselect the candidate medium 115 based on the concept predictions 309from the reviewers 209. For example, media 115 may be selected where theconcept prediction 309 is close to 0.5, indicating that the review model105 is uncertain of an appropriate concept. In one embodiment, media 115with concept predictions 309 that satisfy the objective description 305are selected for review. In a certain embodiment, a “human in the loop”such as the expert 205 and/or moderator 207 may assist in selecting themedia 115.

The target strategy selects media 115 based on media metrics 353. Forexample, the processor 405 may select media 115 with a specifiedkeywords. The processor 405 may further select a set of media 115 basedon the media sourcing strategy.

The processor 405 may generate 505 media reviews 371 from a plurality ofreviewers 209 using the review job 301 for the expert media set 113. Inone embodiment, the processor 405 presents the review job 301 to thereviewers 209 for each medium 115 in the reviewer media set 114 andreceives the review 351 including the concept prediction 309 for themedium 115 from each reviewer 209. The concept predictions 309 may beaggregated in the consensus concept prediction 373 of the media review371 for the medium 115. The generation 505 of the media reviews 371 isdescribed in more detail in FIG. 5D.

The processor 405 may further extract 507 the media features 317 for theplurality of media reviews 371. In one embodiment, the processor 405performs an automated review of the media 115 in the reviewer media set114 to extract 507 the media features 317. The extraction 506 of themedia features 317 is described in more detail in FIG. 5H.

The processor 405 may train 509 the review model 105. The processor 405may train 509 the review model 105 with media reviews 371 andcorresponding media features 317. In one embodiment, the review model105 is trained 509 with a training data set of the media features 317and corresponding concept predictions 309 for each medium 115 of thereviewer media set 114. The training 509 of the review model 105 isdescribed in more detail in FIG. 5E.

In one embodiment, the review model 105 is retrained 509 with media 115selected using the media sourcing strategy. The processor 405 mayregularly retrain 509 the review model 105. In a certain embodiment, thereview model 105 is retrained daily.

The processor 405 may select 511 a production review model 105. Theproduction review model 105 may be selected 511 from one or more trainedreview models 105 and/or a current production review model 105. In oneembodiment, each review model 105 is tested against the expert media set113. The review model 105 that predicts concept predictions 309 that areclosest to the expert concept predictions 313 of the review job 301 maybe selected 511.

The processor 405 may verify 513 that the selected production reviewmodel 105 is production ready and the method 500 ends. The selectedproduction review model 105 may be tested against a production subset ofthe media corpus 101 to verify 513 that the selected production reviewmodel 105 is production ready.

FIG. 5B is a flow chart diagram illustrating one embodiment of a mediaselection method 520. The method 520 may select the media set 107 fromthe media corpus 101 using the review model 105. The method 520 may beperformed by the processor 405 and/or neural network 475.

The method 520 starts, and in one embodiment, the processor 405 extracts521 the media features 317 from the media 115 of the media corpus 101.The processor 405 may perform an automated review of the media 115 toextract 521 the media features 317. The extraction 521 of the mediafeatures 317 is described in more detail in FIG. 5H.

The processor 405 may produce 523 the concept prediction 309 using thereview model 105. The concept prediction 309 may be produced 523 Theconcept prediction 309 may be produced 523 by predicting the conceptprediction 309. In one embodiment, the media features 317 for eachmedium 115 are presented to the review model 105 and the review model105 predicts 523 the concept prediction 309 based on the media features317. In a certain embodiment, the production review model 105 isemployed.

The processor 405 may perform 525 a constrained optimization calculationfor presenting the media 115. The constrained optimization calculationmay optimize the objective function 227 for the budget requirements 221,the placement number 223, and the desired spend 225 to generate theconstrained optimization 229. The constrained optimization calculationis described in more detail in FIG. 5F.

The processor 405 may select 527 media 115 based on the predictedconcept prediction 309 for the media set 107. In one embodiment, theselection 527 is based on the concept predictions 309 for the media 115and/or the constraints of the constrained optimization 229. A givenmedium 115 may be selected 527 based on a concept prediction 309 thatthe given medium 115 adheres to the brand's intent. In addition,although the concept prediction 309 may indicate that the given medium115 adheres to the brand's intent, the given medium 115 may not beselected if a forecast cost of the given medium exceeds the constraintsof the constrained optimization 229. One embodiment of the selection 527of the media 115 is described in more detail in FIG. 5G.

The processor 405 may present 529 a promotion 319 with the selectedmedia 115 of the media set 107 and the method 520 ends. In oneembodiment, when the selected media 115 is presented to a user, thepromotion 319 may also be presented. For example, when the user views aYOUTUBE® video that is included in the media set 107, a video promotion319 may also be presented 529.

FIG. 5C is a flow chart diagram illustrating one embodiment of reviewjob generation method 550. The method 550 generates the review job 301.The method 550 may be performed by the processor 405. The method 550starts, and in one embodiment, the processor 405 presents 551 media 115of the expert media set 113 to the expert 205. The media 115 may bepresented 551 during a meeting between the expert 205 and the moderator207. In addition, the media 115 may be presented 551 during an onlineinteraction.

The processor 405 may capture 553 the expert concept prediction 313 inresponse to presenting 551 the media 115. The processor 405 may querythe expert 205 with the review question 307 regarding the expert'sopinions of the media 115. The expert's opinions may be recorded as theexpert concept prediction 313. In addition, the processor 405 may recordany initial views on concepts that support a brand that the expert 205has as the objective description 305. The processor 405 may furtherquery the expert 205 regarding the expert's reasons for the opinions.The reasons may be recorded as the answer description 315.

The processor 405 may standardize 555 the preference format 340 for theexpert concept prediction 313. The standardization 555 of the expert'sresponses using the preference format 340 supports the efficient andaccurate generation of the review job 301 by the computer 400. Thus, thepreference format 340 enables the automated generation of the review job301. In one embodiment, each review question 307 that elicits an expertconcept prediction 313 is collated into a review set 261 along with thecorresponding objective description 305, expert concept prediction 313,and/or answer description 315.

In one embodiment, the expert concept prediction 313 is tested forsentiment. The review question 307 may be determined to have elicitedthe expert concept prediction 313 if the sentiment of the expert conceptprediction 313 exceeds the sentiment threshold 394. The inclusion of areview question 307 and corresponding review set 261 in the preferenceformat 340 based on sentiment improves the accuracy of the preferenceformat 340 in reflecting the expert's opinions regarding the media 115.As a result, the resulting review job 301 is more efficient in gatheringreviews 351.

The processor 405 may design 557 the review job 301 and/or review jobdatabase 300 based on the plurality of expert concept predictions 313 inthe preference format 340. In one embodiment, the review job 301 and/orreview job database 300 is based on at least a minimum prediction numberof expert concept predictions 313 and/or review jobs 301. The minimumprediction number may be in the range of 50 to 1000. In a certainembodiment, the minimum prediction number is 100.

In one embodiment, the minimum prediction number of expert conceptpredictions 313 include a minimum positive number of expert conceptpredictions 313 with a positive review and a minimum negative number ofexpert concept predictions 313 with a negative review. The minimumpositive number and the minimum negative number may be in the range of30 to 50. The processor 405 may design 557 the review job 301 byselecting the minimum prediction number of review sets 261 andgenerating corresponding review jobs 301 for each review set 261.

The processor 405 presents 559 the review job 301 to a plurality ofreviewers 209 for the expert media set 113. In one embodiment, at least30 reviewers 209 are presented 559 with the review job 301. In a certainembodiment, the processor 405 presents each medium 115 of the expertmedia set 113 to the reviewers 209 along with the corresponding reviewjob 301. For example, the processor 405 may present 559 a first medium115 and the review job 301 with the media identifier 303 for the firstmedium 115. In a certain embodiment, the first medium 115 is presented559 to the reviewer 209 and the review question 307 is presented 559 tothe reviewer 209. In addition, one or more of the objective description305, the test question 311, the expert concept prediction 313, theanswer description 315, the media features 317, and/or the promotion 319may be presented 559 to the reviewer 209.

The reviewers 209 answer the review question 307 to the best of theirability. The processor 405 receives 561 the answer as the conceptprediction 309. The concept prediction 309 may be stored with the mediaidentifier 303 in a review 351.

The processor 405 may determine 563 whether the agreement threshold 395is satisfied for concept predictions 309 received from the reviewers209. The agreement threshold 395 may be satisfied if agreement betweenthe concept predictions 309 received from the reviewers 209 and theexpert concept predictions 313 is statistically significant. Inaddition, the agreement threshold 395 may be satisfied if agreementbetween the concept predictions 309 received from the reviewers 209 andthe expert concept predictions 313 satisfies a Bayesian predictiveprobability of success.

In one embodiment, the reviewers 209 are asked the test question 311 todetermine if the reviewers 209 understand the experts' opinions. Shoulda reviewer 209 fail to get a sufficient number of test questions 311correct, the reviewer 209 is removed from the set of reviewers 209 andthe reviewer's reviews 351 are removed from the review database 350.

If the agreement threshold 395 is not satisfied, the processor 405 mayredesign 557 the review job 301 and/or review job database 300. In oneembodiment, review job instances 301 with a lowest match betweenreviewer concept predictions 309 an expert concept predictions 313 maybe removed from the review job 301. The review job 301 may beiteratively redesigned 557 by the processor 405 until the agreementthreshold 395 is satisfied.

In response to the agreement threshold 395 being satisfied, the method550 ends. The review job 301 is validated as reflective of the expertconcept predictions 313 of the experts 205 and is ready for deploymentto additional reviewers 209 reviewing media 115 from the media corpus101. The automated design of the review job 301 and review job database300 uses the preference format 240 to generate review jobs 301 thataccurately reflect expert opinions. As a result, the computer 400 isenabled to generate review jobs 301 used for the training of the reviewmodel 105.

FIG. 5D is a flow chart diagram illustrating one embodiment of mediareview generation method 570. The method 570 may record media reviews371 using the validated review job 301. The method 570 may be performedby the processor 405.

The method 570 starts, and in one embodiment, the processor 405identifies 571 a plurality of reviewers 209. The plurality of reviewers209 may be identified 571 based on concept predictions 309 for thereview jobs 301 for the expert media set 113. For example, a givenreviewer 209 may be identified 571 if the given reviewer's conceptpredictions 309 satisfy the accuracy threshold 393 when compared withthe expert concept predictions 313 for media 115 of the expert media set113.

The processor 405 further presents 573 the review job 301 to theplurality of reviewers 209. The review job 301 may be presented 573 bydisplaying the medium 115 identified by the media identifier 303 and bypresenting 573 the review question 307. The medium 115 may be selectedfrom the review media set 114 of the media corpus 101. In a certainembodiment, the medium 115 is selected from the expert media set 113. Inaddition, one or more of the objective description 305, the testquestion 311, the expert concept prediction 313, the answer description315, the media features 317, and/or the promotion 319 may be presented573.

The processor 405 may receive 575 the concept prediction 309 for thereview question 307 of the review job 301 from the plurality ofreviewers 209 in response to the medium 115 and the review question 307.The media identifier 303 and concept prediction 309 may be stored as areview 351.

The processor 405 may retrieve 577 the plurality of reviews 351 from thereview database 350 and determine 579 if concept predictions 309 satisfythe accuracy threshold 393. The accuracy threshold 393 may be satisfiedif the probability of a null hypothesis is less than the accuracythreshold 393. In addition, the agreement threshold 395 may be satisfiedif a probability given a prior is greater than the accuracy threshold393. In one embodiment, the accuracy threshold 393 is satisfied if asufficiently large set of reviewers 209 agree on the concept prediction309.

In one embodiment, each medium 115 is presented 573 to a small number ofreviewers 209. Should the accuracy threshold 393 not be satisfied by fewreviewers 209, the same medium 115 may be presented to more reviewers209 up until a maximum number of reviewers 209 per medium 115 isreached.

If the accuracy threshold 393 is satisfied for the plurality reviewers209, the processor 405 may record 587 the media review 371 and themethod 570 ends. The media review 371 may be recorded 587 with theconsensus concept prediction 373. The consensus concept prediction 373may be an average of the concept predictions 309 from the reviewers 209,a mean of the concept predictions 309, a median of the conceptpredictions 309, and/or a random sample of the concept predictions 309.

If the accuracy threshold 393 is not satisfied for the plurality ofreviewers 209, the processor 405 may remove reviewers 209 from theplurality of reviewers 209. The removed reviewers 209 may have madeconcept predictions 309 that do not satisfy the accuracy threshold 393.In one embodiment, the removed reviewers 209 made concept predictions309 that did not agree with the expert concept predictions 313.

The processor 405 may further determine 583 if the accuracy threshold393 is satisfied for the remaining plurality of reviewers 209. In oneembodiment, the processor 405 determines 583 if the accuracy threshold393 is satisfied for the remaining reviewers 209 after reviewers 209with concept predictions 309 that did not agree with the expert conceptpredictions 313 were removed from the plurality of reviewers 209. If theaccuracy threshold 393 is still not satisfied, the processor 405 mayregenerate 585 the review job 301 and present 573 the regenerated reviewjob 301 to the reviewers 209. The regeneration 585 of the review job 301may employ the method 550 of FIG. 5C.

If the accuracy threshold 393 is satisfied, the processor 405 may record587 the media review 371 and the method 570 ends. The method 570generates media reviews 371 with concept predictions 309 that anticipatethe expert concept predictions 313. In addition, the use of theplurality of reviewers 209 allows the recording of sufficient mediareviews 371 to train the review model 105 as will be describedhereafter.

FIG. 5E is a flow chart diagram illustrating one embodiment of a modeltraining method 600. The method 600 trains the review model 105 with themedia reviews 371. The method 600 may be performed by the processor 405and/or neural network 475 and/or gradient boosting machines.

The method 600 starts, and in one embodiment, the processor 405 extracts601 the media features 317 for the media 115 of the media review set114. The media features 317 may be regularly and/or automaticallyextracted 601. In a certain embodiment, the media features 317 areautomatically extracted 601 daily. In one embodiment, the processor 405encodes the media category 247, the text description 249, the image 251,the text 253, the video 255, and/or the audio 257. The extraction 601 ofthe media features 317 is described in more detail in FIG. 5H.

The processor 405 may modify 603 the model parameters 391 for the reviewmodel 105. In one embodiment, the modification 603 of the modelparameters 391 is directed towards a local optimum. In addition, themodification 603 may be a random perturbation of the model parameters391. The processor 405 may modify 603 a model type, a model breadth, amodel depth, a learning rate, a learning algorithm, a network geometry,and/or a recursion geometry.

The processor 405 may further train 605 a given review model 105 basedon the modified model parameters 391. The processor 405 may iterativelyapply a learning algorithm to the media reviews 371 for a specifiedmodel type based on the model parameters 391.

The processor 405 may compare 607 the given review model 105 to acurrent review model 105 if the current review model 105 exists. In acertain embodiment, both the given review model 105 and the currentreview model 105 make predictions of the concept prediction 309 for theexpert media set 113. The processor 405 may determine 609 the givenreview model 105 is an improvement if the concept predictions 309 of thegiven review model 105 are in closer agreement to the expert conceptpredictions 313 for the expert media set 113 than the conceptpredictions 309 of the current review model 105.

If the given review model 105 is not an improvement, the processor 405again modifies 603 the model parameters 391. The modification 603 may bebased on differences between the given review model 105 and the currentreview model 105. If the given review model 105 is an improvement, theprocessor 405 determines 611 if optimization of the model parameters 391is complete. In one embodiment, optimization of the model parameters 391is complete if repeated perturbations of the model parameters 391converge on the model parameters 391 for the given review model 105.

If the optimization is not complete, the processor 405 again modifies603 the model parameters 391. If the optimization is complete, theprocessor 405 selects 613 the current review model 105 as the trained,optimized review model 105 and the method 600 ends. The method 600trains and selects the review model 105 that accurately predicts theconcept prediction 309 based on the media reviews 371.

FIG. 5F is a flow chart diagram illustrating one embodiment ofconstrained optimization method 650. The method 650 performs aconstrained optimization calculation for selecting media 115. The method650 may be performed by the processor 405.

The method 650 starts, and in one embodiment, the processor 405 defines651 the budget requirements 221 for presenting the promotion 319 withthe media 115. The budget requirements 221 may be received from theadvertiser. The processor 405 further defines 653 the objective function227 for presenting the media 115. The processor 405 may define 655 theconstraints for the objective function 227. The constraints may be theplacement number 223 and/or the desired spend 225 from the budgetrequirements 221. The processor 405 may further calculate 657 theconstrained optimization 227 and the method 650 ends. The constrainedoptimization 227 may be employed to determine the media 115 for themedia set 107 and/or the medium bid 123.

FIG. 5G is a flow chart diagram illustrating one embodiment of mediaselection method 700. The method 700 may select media 115 for the mediaset 107 based on predicted concept predictions 309. In a certainembodiment, the method 700 determines whether to replace current media115 with new media 115 in the media set 107. The promotion 319 may bepresented to users along with the media 115 of the media set 107. Themethod 700 may be performed by the processor 405 and/or the neuralnetwork 475 and/or gradient boosting machine.

The method 700 starts, and in one embodiment, the processor 405 predicts701 the concept prediction 309 for current media 115 in the media set107. The current media 115 may be already included in the media set 107.The processor 405 may employ the review model 105 to predict 701 theconcept prediction 309. The concept prediction 309 may be predicted 701by applying the media features 317 to the review model 105 and receivingthe concept prediction 309 in response to the media features 317.

The processor 405 further predict 703 the concept prediction 309 for thenew media 115 using the review model 105. If the concept prediction 309for the new media 115 indicates that the new media 115 better satisfiesthe advertiser's brand intent, the processor 405 replaces 705 thecurrent media 115 with the new media 115 and the method 700 ends. Themethod 700 uses the review model 105 to build the media set 107 of media115 that satisfies the advertiser's brand intent.

FIG. 5H is a flow chart diagram illustrating one embodiment of mediafeature identification method 720. The method 720 extracts the mediafeatures 317 from media 115. The method 720 may be performed by theprocessor 405 and/or the neural network 475 and/or the gradient boostingmachine.

The method 720 starts, and in one embodiment, the processor 405 receives721 the media metrics 353 for a medium 115. The media metrics 353 may bedownloaded from a provider of the medium 115. For example, the mediametrics 353 may be downloaded from YOUTUBE® for a YOUTUBE® video medium115.

The processor 405 may further encode 723 the media category 247 for themedium 115. In one embodiment, the processor 405 employs the neuralnetwork 475 to generate a learned embedding from one or more of themedia metrics 353, text description 249, image 251, text 253, video 255,and/or audio 257. The media category 247 may be based on the learnedembedding. The media category 247 also may be encoded 723 based onkeywords from the media metrics 373.

The processor 405 may encode 725 the text description 249 for the medium115. The processor 405 may employ the neural network 475 to generate alearned embedding from the text 253, video 255, and/or audio 257. Thetext description 249 may be based on the learned embedding.

In one embodiment, the processor 405 and/or neural network 475 comparesthe image 251, the text 253, the video 255, and/or the audio 257 to theimage 251, the text 253, the video 255, and/or the audio 257 of themedia features 317 of other media 115 with the text description 249 ofthe other media 115. The processor 405 may apply the text description249 of the other media 115 with images 251, text 253, video 255, and/oraudio 257 matching the image 251, the text 253, the video 255, and/orthe audio 257 of the medium 115.

The processor 405 and/or neural network 475 may encode 727 the image 251for the medium 115. The processor 405 and/or neural network 475 maycompare the media metrics 353, media category 247, text description 249,text 253, video 255, and/or audio 257 of the other media 115 with themedia metrics 253, media category 247, text description 249, text 253,video 255, and/or audio 257 of the medium 115. The processor 405 and/orneural network 475 may further encode 727 the image 251 based on theimage 251 of the other media 115 with media metrics 353, media category247, text description 249, text 253, video 255, and/or audio 257 thatmostly closely match the media metrics 353, media category 247, textdescription 249, text 253, video 255, and/or audio 257 of the medium115.

The processor 405 and/or neural network 475 may encode 729 the text 253for the medium 115. The neural network 475 may generate a learnedembedding from the video 255 and/or audio 257. The text 253 may be basedon the learned embedding.

In one embodiment, the processor 405 and/or neural network 475 maycompare the media metrics 353, media category 247, text description 249,image 251, video 255, and/or audio 257 of other media 115 with the mediametrics 253, media category 247, text description 249, image 251, video255, and/or audio 257 of the medium 115. The processor 405 and/or neuralnetwork 475 may further encode 729 the text 253 based on the text 253 ofthe other media 115 with media metrics 353, media category 247, textdescription 249, image 251, video 255, and/or audio 257 that mostlyclosely match the media metrics 353, media category 247, textdescription 249, image 251, video 255, and/or audio 257 of the medium115.

The processor 405 and/or neural network 475 may encode 731 the video 255of the media 115 and the method 720 ends. In one embodiment, the neuralnetwork 475 generates a learned embedding from the audio 257. The video255 may be based on the learned embedding.

In one embodiment, the processor 405 and/or neural network 475 maycompare the media metrics 353, media category 247, text description 249,image 251, text 253, video 255, and/or audio 257 of other media 115 withthe media metrics 253, media category 247, text description 249, image251, text 253, video 255, and/or audio 257 of the medium 115. Theprocessor 405 and/or neural network 475 may further encode 731 the video255 based on the video 255 of other media 115 with media metrics 353,media category 247, text description 249, image 251, text 255, and/oraudio 257 that mostly closely match the media metrics 353, mediacategory 247, text description 249, image 251, text 253, and/or audio257 of the medium 115.

FIG. 5I is a flow chart diagram illustrating one embodiment of mediabidding method 750. The method 750 may generate the medium bid 123. Themethod 750 may be performed by the processor 405 and/or the neuralnetwork 475.

The method 750 starts, and the processor 405 receives a target medium115 that is scheduled for consumption. For example, the target medium115 may be a video medium 115 that may be viewed by a user. Theprocessor 405 may further extract 753 the media features 317 from thetarget medium 115. The media features 317 may be extracted 753 asdescribed in FIG. 5H.

The processor 405 further predicts 755 the concept prediction 309 forthe target medium 115. In one embodiment, the processor 405 applies themedia features 317 of the target medium 115 to the review model 105. Theprocessor 405 further receives the concept prediction 309 from thereview model 105.

The processor 405 calculate 757 a bid for the target medium 115 based onthe concept prediction 309. As a result, the processor 405 is able tobid higher for target media 115 that satisfies and advertisers brandintent, enhancing the calculation of the bid.

The embodiments automate the generation and validation of a review job301 that enables a large number of reviewers 209 to review a largereviewer media set 114, generating sufficient reviews 350 for trainingthe review model 105. The review model 105 makes concept predictions 309that closely match the expert concept predictions 313, enabling a largenumber of media 115 from the media corpus 101 to be efficiently andaccurately reviewed by the computer 400, improving the efficiency of thecomputer 400.

Embodiments may be practiced in other specific forms. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A method comprising: generating, by use of aprocessor, a review job comprising at least one review question and acorresponding expert concept prediction for an expert media set that isa subset of a media corpus; generating media reviews from a plurality ofreviewers using the review job for the expert media set; extractingmedia features for media of the media reviews; training a review modelwith the media reviews and corresponding media features; producing theconcept prediction for media of the media corpus using the review model;and selecting media based on the predicted concept prediction for amedia set.
 2. The method of claim 1, wherein generating the review jobcomprises: presenting media of the expert media set to an expert;capturing the expert concept prediction; standardizing a preferenceformat for the expert concept prediction; and designing the review jobbased on a plurality of expert concept predictions.
 3. The method ofclaim 2, wherein generating the review job further comprises: presentingthe review job to a plurality of reviewers; determining whether anagreement threshold is satisfied for concept predictions received fromthe reviewers; and redesigning the review job in response to theagreement threshold not being satisfied.
 4. The method of claim 1,wherein generating the media reviews comprises: presenting the reviewjob to the plurality of reviewers; receiving concept predictions for thereview question of the review job from the plurality of reviewers;determining whether the concept predictions satisfy an accuracythreshold; and recording the media review in response to the accuracythreshold being satisfied, wherein each media review comprises aconsensus concept prediction.
 5. The method of claim 4, whereingenerating the media review further comprises: regenerating the reviewjob in response to the accuracy threshold not being satisfied for theplurality of reviewers; and removing reviewers from the plurality ofreviewers with concept predictions that do not satisfy the accuracythreshold.
 6. The method of claim 4, wherein generating the mediareviews further comprises identifying a plurality of reviewers basedanswers to the review job for the expert media set from the plurality ofreviewers.
 7. The method of claim 1, wherein generating the training thereview model comprises: extracting media features from media; modifyingmodel parameters for the review model; and training a given review modelbased on the modified model parameters; comparing the given review modelto a current review model; and selecting the given review model as thecurrent review model in response to improvement.
 8. The method of claim1, the method further comprising performing a constrained optimizationcalculation for presenting the media.
 9. The method of claim 8, whereinperforming the constrained optimization calculation comprises: definingbudget requirements for presenting the media; defining an objectivefunction for presenting the media; defining constraints comprising aplacement number and a desired spend from the budget requirements; andcalculating the constrained optimization.
 10. The method of claim 1, themethod further comprising presenting a promotion with the selectedmedia.
 11. The method of claim 1, the method further comprising:receiving a target medium; predicting the concept prediction for thetarget medium; and calculating a bid for the target medium based on theconcept prediction.
 12. The method of claim 1, the method further:selecting a media sourcing strategy from the group consisting of arandom strategy, an active learning strategy, and a targeted strategy,wherein the random strategy selects media at random, the active learningstrategy selects media based on the concept prediction and interactivequeries directed to reviewers, and the target strategy selects mediabased on media metrics; and retraining the review model with mediaselected using the media sourcing strategy.
 13. The method of claim 1,wherein extracting media features comprises: receiving media metrics formedia; encoding a media category for the media; encoding a textdescription for the media; encoding an image for the media; encodingtext from audio of the media; and encoding video of the media.
 14. Anapparatus comprising: a processor; a memory that stores code executableby the processor to perform: generating a review job comprising at leastone review question and a corresponding expert concept prediction for anexpert media set that is a subset of a media corpus; generating mediareviews from a plurality of reviewers using the review job for theexpert media set; extracting media features for media of the mediareviews; training a review model with the media reviews andcorresponding media features; producing the concept prediction for mediaof the media corpus using the review model; and selecting media based onthe predicted concept prediction for a media set.
 15. The apparatus ofclaim 14, wherein generating the review job comprises: presenting mediaof the expert media set to an expert; capturing the expert conceptprediction; standardizing a preference format for the expert conceptprediction; and designing the review job based on a plurality of expertconcept predictions.
 16. The apparatus of claim 15, wherein generatingthe review job further comprises: presenting the review job to aplurality of reviewers; determining whether an agreement threshold issatisfied for concept predictions received from the reviewers; andredesigning the review job in response to the agreement threshold notbeing satisfied.
 17. The apparatus of claim 14, wherein generating themedia reviews comprises: presenting the review job to the plurality ofreviewers; receiving concept predictions for the review question of thereview job from the plurality of reviewers; determining whether theconcept predictions satisfy an accuracy threshold; and recording themedia review in response to the accuracy threshold being satisfied,wherein each media review comprises a consensus concept prediction. 18.The apparatus of claim 17, wherein generating the media review furthercomprises: regenerating the review job in response to the accuracythreshold not being satisfied for the plurality of reviewers; andremoving reviewers from the plurality of reviewers with conceptpredictions that do not satisfy the accuracy threshold.
 19. Theapparatus of claim 17, wherein generating the media reviews furthercomprises identifying a plurality of reviewers based answers to thereview job for the expert media set from the plurality of reviewers. 20.A program product comprising a non-transitory storage medium storingcode executable by the processor to perform: generating a review jobcomprising at least one review question and a corresponding expertconcept prediction for an expert media set that is a subset of a mediacorpus; generating media reviews from a plurality of reviewers using thereview job for the expert media set; extracting media features for mediaof the media reviews; training a review model with the media reviews andcorresponding media features; producing the concept prediction for mediaof the media corpus using the review model; and selecting media based onthe predicted concept prediction for a media set.